Evaluation of digital twins using social automatons

ABSTRACT

In one example, the present disclosure improves simulations based on the use of digital twins using social automatons. In one example, a method performed by a processing system including at least one processor includes constructing a digital twin of a physical environment, constructing a social automaton, wherein the social automaton comprises a virtual representation of an individual that is programmed to exhibit behaviors and characteristics of the individual, and inserting the social automaton into the digital twin to create an extended reality media in which interactions of the social automaton with the digital twin simulate interactions of the individual with the physical environment.

This application is a continuation of U.S. patent application Ser. No.16/747,780, filed Jan. 21, 2020, now U.S. Pat. No. 11,068,046, which isherein incorporated by reference in its entirety.

The present disclosure relates generally to extended reality (XR) media,and relates more particularly to devices, non-transitorycomputer-readable media, and methods for evaluating digital twins usingsocial automatons.

BACKGROUND

Extended reality (XR) is an umbrella term used to describe various typesof immersive technology, including augmented reality (AR), virtualreality (VR), and mixed reality (MR), in which the real-worldenvironment may be enhanced or augmented with virtual,computer-generated objects or actions. One particular use of XRtechnology involves the creation of digital “twins,” or virtual modelsof real, physical items. The ability to create a digital twin of aphysical environment, for example, has become an increasingly powerfulalternative to costly buildout for understanding the impacts of designand placement changes. For instance, in an industrial setting, a digitaltwin of a manufacturing floor may be constructed in order to assess theimpacts of proposed layout changes (e.g., relocation of equipment,egress and ingress points, etc.). In real estate applications, a digitaltwin of a room in a home may be constructed to evaluate differentarrangements of furniture or different architectural changes (e.g.,moving a wall, installing cabinetry, etc.). In urban planningapplications, a digital twin of a road intersection may be constructedin order to evaluate different changes to the intersection'sconfiguration (e.g., left turn lane versus jug handle versus roundabout,etc.).

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present disclosure can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 illustrates an example network related to the present disclosure;

FIG. 2 illustrates a flowchart of a method for evaluating a digital twinof a physical environment using a social automaton, in accordance withthe present disclosure;

FIG. 3A illustrates an example digital twin that may be constructedaccording to the method of FIG. 2;

FIG. 3B illustrates an example overlay comprising set of socialautomatons that may be constructed for insertion into the exampledigital twin of FIG. 3A;

FIG. 3C illustrates an extended reality media in which the socialautomatons of FIG. 3B may be inserted into the digital twin of FIG. 3A;and

FIG. 4 depicts a high-level block diagram of a computing devicespecifically programmed to perform the functions described herein.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

DETAILED DESCRIPTION

In one example, the present disclosure improves simulations based on theuse of digital twins using social automatons. In one example, a methodperformed by a processing system including at least one processorincludes constructing a digital twin of a physical environment,constructing a social automaton, wherein the social automaton comprisesa virtual representation of an individual that is programmed to exhibitbehaviors and characteristics of the individual, and inserting thesocial automaton into the digital twin to create an extended realitymedia in which interactions of the social automaton with the digitaltwin simulate interactions of the individual with the physicalenvironment.

In another example, a non-transitory computer-readable medium storesinstructions which, when executed by a processing system including atleast one processor, cause the processing system to perform operations.The operations include constructing a digital twin of a physicalenvironment, constructing a social automaton, wherein the socialautomaton comprises a virtual representation of an individual that isprogrammed to exhibit behaviors and characteristics of the individual,and inserting the social automaton into the digital twin to create anextended reality media in which interactions of the social automatonwith the digital twin simulate interactions of the individual with thephysical environment.

In another example, a device includes a processor and a non-transitorycomputer-readable medium that stores instructions which, when executedby the processor, cause the processing system to perform operations. Theoperations include constructing a digital twin of a physicalenvironment, constructing a social automaton, wherein the socialautomaton comprises a virtual representation of an individual that isprogrammed to exhibit behaviors and characteristics of the individual,and inserting the social automaton into the digital twin to create anextended reality media in which interactions of the social automatonwith the digital twin simulate interactions of the individual with thephysical environment.

As discussed above, the ability to create a digital twin of a physicalenvironment has become an increasingly powerful alternative to costlybuildout for understanding the impacts of design and placement changes.Proposed changes to the physical environment can be visualized andevaluated in a detailed, scale simulation before the changes areactually implemented in the physical environment. For instance, in anindustrial setting, a digital twin of a manufacturing floor may beconstructed in order to assess the impacts of proposed layout changes(e.g., relocation of equipment, relocation of egress and ingress points,etc.). In real estate applications, a digital twin of a room in a homemay be constructed in order to evaluate different arrangements offurniture or different architectural changes (e.g., moving a wall,installing cabinetry, etc.). In urban planning applications, a digitaltwin of a road intersection may be constructed in order to evaluatedifferent changes to the intersection's configuration (e.g., left turnlane versus jug handle versus roundabout, etc.).

Conventional digital twins of physical environments tend to simulate thestatic elements of the physical environments well. For instance,techniques such as scanning with three-dimensional object recognitionand placement can be used to create a detailed digital twin thatreplicates the boundaries (e.g., walls, floors, ceilings, etc.) of aphysical environment, the objects (e.g., furniture, machinery, etc.)that are present in the physical environment, and other static elements.However, the more dynamic, social (e.g., human) elements of the physicalenvironments are not as easy to replicate. For instance, it may beuseful to simulate the effects that changes in the physical environmentmay have on human interactions and human-driven events in the physicalspace (e.g., fire drills, meetings, parties, etc.). As an example,constructing a digital twin of a room may help to confirm that a largepiece of furniture will fit within the walls of the room. However, thepiece of furniture may partially obstruct human access to another itemin the room or may create a bottleneck when many people are present inthe room, and these are effects that a simple static simulation may failto capture.

Although more dynamic elements can be simulated in the digital twin,conventional approaches to simulating the dynamic elements tend to belargely computational. As an example, traffic conditions at a roadintersection at a given time of day may be simulated based on historicaltraffic information for the intersection, which may be obtained from adatabase. For instance, the database may indicate an average number ofvehicles that pass through the intersection in each direction during agiven window of time (e.g., Mondays between 7:00 AM and 10:00 AM). Thesesimulations may fail to capture natural variations in behavior,anomalies, and the like.

Examples of the present disclosure provide social automatons forevaluating digital twins of physical environments. Within the context ofthe present disclosure, a “social automaton” is understood to be avirtual representation of an individual, such as a person (e.g., aspecific person or a representative of a particular demographic) or ananimal (e.g., a cat or dog). The social automaton may be deployed withinan XR environment, such as a digital twin. The social automaton in thiscontext may be programmed to demonstrate different behaviors and toexhibit different characteristics that a real individual in the physicalenvironment might demonstrate. Moreover, the ways in which differentsocial automatons interact with each other in the digital twin canindicate how interactions between real individuals may occur in thecorresponding physical environment.

The social automatons may thus be deployed into the digital twin of thephysical environment, which may be altered (relative to the actualphysical environment) to incorporate some proposed change to thephysical environment. The social automatons may explore the digital twinin an interactive manner, and the reactions of the social automatons tothe digital twin and to each other while in the digital twin may beobserved. In this way, the dynamic impacts of changes to the physicalenvironment can be evaluated before the changes are implemented,allowing for improvements to the changes to be made preemptively.Moreover, using a social automaton rather than a computationalsimulation may provide for more natural interactions to be simulated inthe digital twin, thereby allowing for a more realistic understanding ofthe impacts of various changes in the physical environment.

To better understand the present disclosure, FIG. 1 illustrates anexample network 100 related to the present disclosure. As shown in FIG.1, the network 100 connects mobile devices 157A, 157B, 167A and 167B,and home network devices such as home gateway 161, set-top boxes (STBs)162A, and 162B, television (TV) 163A and TV 163B, home phone 164, router165, personal computer (PC) 166, and so forth, with one another and withvarious other devices via a core network 110, a wireless access network150 (e.g., a cellular network), an access network 120, other networks140 and/or the Internet 145.

In one example, wireless access network 150 comprises a radio accessnetwork implementing such technologies as: global system for mobilecommunication (GSM), e.g., a base station subsystem (BSS), or IS-95, auniversal mobile telecommunications system (UMTS) network employingwideband code division multiple access (WCDMA), or a CDMA3000 network,among others. In other words, wireless access network 150 may comprisean access network in accordance with any “second generation” (2G),“third generation” (3G), “fourth generation” (4G), Long Term Evolution(LTE) or any other yet to be developed future wireless/cellular networktechnology including “fifth generation” (5G) and further generations.While the present disclosure is not limited to any particular type ofwireless access network, in the illustrative example, wireless accessnetwork 150 is shown as a UMTS terrestrial radio access network (UTRAN)subsystem. Thus, elements 152 and 153 may each comprise a Node B orevolved Node B (eNodeB).

In one example, each of mobile devices 157A, 157B, 167A, and 167B maycomprise any subscriber/customer endpoint device configured for wirelesscommunication such as a laptop computer, a Wi-Fi device, a PersonalDigital Assistant (PDA), a mobile phone, a smartphone, an email device,a computing tablet, a messaging device, a wearable smart device (e.g., asmart watch or fitness tracker), a gaming console, and the like. In oneexample, any one or more of mobile devices 157A, 157B, 167A, and 167Bmay have both cellular and non-cellular access capabilities and mayfurther have wired communication and networking capabilities.

As illustrated in FIG. 1, network 100 includes a core network 110. Inone example, core network 110 may combine core network components of acellular network with components of a triple play service network; wheretriple play services include telephone services, Internet services andtelevision services to subscribers. For example, core network 110 mayfunctionally comprise a fixed mobile convergence (FMC) network, e.g., anIP Multimedia Subsystem (IMS) network. In addition, core network 110 mayfunctionally comprise a telephony network, e.g., an InternetProtocol/Multi-Protocol Label Switching (IP/MPLS) backbone networkutilizing Session Initiation Protocol (SIP) for circuit-switched andVoice over Internet Protocol (VoIP) telephony services. Core network 110may also further comprise a broadcast television network, e.g., atraditional cable provider network or an Internet Protocol Television(IPTV) network, as well as an Internet Service Provider (ISP) network.The network elements 111A-111D may serve as gateway servers or edgerouters to interconnect the core network 110 with other networks 140(which may include servers 149), Internet 145, wireless access network150, access network 120, and so forth. As shown in FIG. 1, core network110 may also include a plurality of television (TV) servers 112, aplurality of content servers 113, a plurality of application servers114, an advertising server (AS) 117, and an extended reality (XR) server115 (e.g., an application server). For ease of illustration, variousadditional elements of core network 110 are omitted from FIG. 1.

With respect to television service provider functions, core network 110may include one or more television servers 112 for the delivery oftelevision content, e.g., a broadcast server, a cable head-end, and soforth. For example, core network 110 may comprise a video super huboffice, a video hub office and/or a service office/central office. Inthis regard, television servers 112 may interact with content servers113, advertising server 117, and XR server 115 to select which videoprograms, or other content and advertisements to provide to the homenetwork 160 and to others.

In one example, content servers 113 may store scheduled televisionbroadcast content for a number of television channels, video-on-demandprogramming, local programming content, gaming content, and so forth.The content servers 113 may also store other types of media that are notaudio/video in nature, such as audio-only media (e.g., music, audiobooks, podcasts, or the like) or video-only media (e.g., imageslideshows). For example, content providers may upload various contentsto the core network to be distributed to various subscribers.Alternatively, or in addition, content providers may stream variouscontents to the core network for distribution to various subscribers,e.g., for live content, such as news programming, sporting events, andthe like. In one example, advertising server 117 stores a number ofadvertisements that can be selected for presentation to viewers, e.g.,in the home network 160 and at other downstream viewing locations. Forexample, advertisers may upload various advertising content to the corenetwork 110 to be distributed to various viewers.

In one example, XR server 115 may generate computer-generated contentincluding digital twins of physical environments. As discussed above, adigital twin of a physical environment comprises a virtual model orreplica of a real world physical environment. For instance, the XRserver 115 may host an application that performs scanning andthree-dimensional object recognition in the physical environment. Theapplication may use the results of the scanning and object recognitionto construct a digital twin that replicates the boundaries (e.g., walls,floors, ceilings, etc.) of the physical environment, the objects (e.g.,furniture, machinery, etc.) that are present in the physicalenvironment, and other elements of the physical environment.Alternatively, the application may retrieve stored information about thedimensions of the physical environment and the dimensions and locationsof objects in the physical environment, and may construct the digitaltwin based on the stored information. In one example, the XR server 115may store the information about the dimensions and locations. In anotherexample, the information may be provided to the XR server 115 by theusers, e.g., via the mobile devices 157A, 157B, 167A, and 167B, the PC166, the home phone 164, the TVs 163A and 163B, and/or Internet ofThings (IoT) devices 168A and 168B. Alternatively, the information maybe retrieved by the XR server 115 from network storage, e.g.,application servers 114.

In a further example, the application hosted on the XR server 115 mayalso generate at least one social automaton for deployment in thedigital twin. As discussed above, the social automaton may comprise avirtual representation of a person (e.g., an avatar) which may bedeployed within an XR environment. The social automaton may represent aspecific person or may be representative of a particular demographic(e.g., a person over the age of sixty-five, a toddler, a person in awheelchair, etc.). The social automaton may be programmed to demonstratedifferent behaviors and to exhibit different characteristics based onthe person or demographic that the social automaton is intended torepresent. For instance, if the social automaton is programmed torepresent a toddler, then the social automaton may be short and unsteadyon its feet and may move quickly. If the social automaton is programmedto represent an elderly person, however, the social automaton may betaller and may move more cautiously.

The application hosted on the XR server 115 may insert the socialautomaton (and potentially additional social automatons) into thedigital twin, and may subsequently simulate the ways in which the socialautomaton interacts with the (replicated) physical environment and withother social automatons in the physical environment. The simulation ofthe social automaton's interactions may be generated at least in partfrom historical or statistical data that indicates typical behaviors forthe person or demographic represented by the social automaton. In oneexample, the XR server 115 may store the historical data. In anotherexample, the historical data may be provided to the XR server 115 by theusers, e.g., via the mobile devices 157A, 157B, 167A, and 167B, the PC166, the home phone 164, the TVs 163A and 163B, and/or Internet ofThings (IoT) devices 168A and 168B. Alternatively, the data may beretrieved by the XR server 115 from network storage, e.g., applicationservers 114.

For instance the historical data may comprise user profiles maintainedby a network service (e.g., an Internet service provider, a streamingmedia service, a gaming subscription, etc.), portions of social mediaprofiles maintained by a social media web site (e.g., a socialnetworking site, a blogging site, a photo-sharing site, etc.), or thelike. The historical data may indicate information about the users, suchas the users' ages, interests, devices (e.g., mobile devices, IoTdevices, gaming devices, etc.), medical or other conditions that mayaffect the users' behaviors and/or mobility in the physical environment,and the like.

The application hosted on the XR server 115 may also insert the socialautomatons into the digital twin in order to simulate events that mayoccur within the physical environment. For instance, the application maydeploy a plurality of social automatons of different demographics inorder to simulate a two hundred person reception in a digital twin ofwedding venue. Alternatively, the application may deploy a plurality ofsocial automatons representing injured and/or ill people in order tosimulate a fire drill in a digital twin of a hospital or assisted livingfacility. As another example, the application may deploy a plurality ofsocial automatons of different demographics in order to simulate theflow of a large crowd entering a digital twin of an amusement park, amuseum, a stadium, or another public space.

In one example, any or all of the television servers 112, contentservers 113, application servers 114, XR server 115, and advertisingserver 117 may comprise a computing system, such as computing system 400depicted in FIG. 4.

In one example, the access network 120 may comprise a Digital SubscriberLine (DSL) network, a broadband cable access network, a Local AreaNetwork (LAN), a cellular or wireless access network, a 3^(rd) partynetwork, and the like. For example, the operator of core network 110 mayprovide a cable television service, an IPTV service, or any other typeof television service to subscribers via access network 120. In thisregard, access network 120 may include a node 122, e.g., a mini-fibernode (MFN), a video-ready access device (VRAD) or the like. However, inanother example node 122 may be omitted, e.g., for fiber-to-the-premises(FTTP) installations. Access network 120 may also transmit and receivecommunications between home network 160 and core network 110 relating tovoice telephone calls, communications with web servers via the Internet145 and/or other networks 140, and so forth.

Alternatively, or in addition, the network 100 may provide televisionservices to home network 160 via satellite broadcast. For instance,ground station 130 may receive television content from televisionservers 112 for uplink transmission to satellite 135. Accordingly,satellite 135 may receive television content from ground station 130 andmay broadcast the television content to satellite receiver 139, e.g., asatellite link terrestrial antenna (including satellite dishes andantennas for downlink communications, or for both downlink and uplinkcommunications), as well as to satellite receivers of other subscriberswithin a coverage area of satellite 135. In one example, satellite 135may be controlled and/or operated by a same network service provider asthe core network 110. In another example, satellite 135 may becontrolled and/or operated by a different entity and may carrytelevision broadcast signals on behalf of the core network 110.

In one example, home network 160 may include a home gateway 161, whichreceives data/communications associated with different types of media,e.g., television, phone, and Internet, and separates thesecommunications for the appropriate devices. The data/communications maybe received via access network 120 and/or via satellite receiver 139,for instance. In one example, television data is forwarded to set-topboxes (STBs)/digital video recorders (DVRs) 162A and 162B to be decoded,recorded, and/or forwarded to television (TV) 163A and TV 163B forpresentation. Similarly, telephone data is sent to and received fromhome phone 164; Internet communications are sent to and received fromrouter 165, which may be capable of both wired and/or wirelesscommunication. In turn, router 165 receives data from and sends data tothe appropriate devices, e.g., personal computer (PC) 166, mobiledevices 167A and 167B, and so forth. In one example, router 165 mayfurther communicate with TV (broadly a display) 163A and/or 163B, e.g.,where one or both of the televisions is a smart TV. In one example,router 165 may comprise a wired Ethernet router and/or an Institute forElectrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi) router, andmay communicate with respective devices in home network 160 via wiredand/or wireless connections.

IoT devices 168A and 168B may include any types of devices that arecapable of being controlled automatically and/or remotely. For instance,the IoT devices 168A and 168B may include “smart” home devices, such asa smart thermostat, a smart lighting system, or the like. The IoTdevices 168A and 168B may also include gaming devices, such as gamingcontrollers, a gaming chair, or the like. Although FIG. 1 illustratestwo IoT devices 168A and 168B, it will be appreciated that the homenetwork 160 may include any number of IoT devices. A greater number andvariety of IoT devices may allow for a more immersive experience to begenerated.

Those skilled in the art will realize that the network 100 may beimplemented in a different form than that which is illustrated in FIG.1, or may be expanded by including additional endpoint devices, accessnetworks, network elements, application servers, etc. without alteringthe scope of the present disclosure. For example, core network 110 isnot limited to an IMS network. Wireless access network 150 is notlimited to a UMTS/UTRAN configuration. Similarly, the present disclosureis not limited to an IP/MPLS network for VoIP telephony services, or anyparticular type of broadcast television network for providing televisionservices, and so forth.

To further aid in understanding the present disclosure, FIG. 2illustrates a flowchart of a method 200 for evaluating a digital twin ofa physical environment using a social automaton, in accordance with thepresent disclosure. In one example, the method 200 may be performed byan XR server that is configured to construct digital twins of physicalenvironments, including social automatons that may be deployed in thedigital twins to simulate human interactions within the physicalenvironment, such as the XR server 115 illustrated in FIG. 1. However,in other examples, the method 200 may be performed by another device,such as the processor 402 of the system 400 illustrated in FIG. 4. Forthe sake of example, the method 200 is described as being performed by aprocessing system.

The method 200 begins in step 202. In step 204, the processing systemmay construct a digital twin of a physical environment. For instance,the physical environment may be a real world location, such as an officebuilding, a school, a factory, an amusement park, a museum, a room in ahouse, or the like. The digital twin may comprise a virtual model thatreplicates the dimensions and layout of the physical environment, aswell as one or more objects that are present in the physicalenvironment. In one example, the digital twin may comprise aninteractive virtual model, such as a virtual model created usingextended reality techniques.

FIG. 3A, for instance, illustrates an example digital twin 300 that maybe constructed according to the method 200 of FIG. 2. In the example ofFIG. 3A, the digital twin replicates the entrance to an amusement park.

The digital twin may be constructed in any one or more of a number ofways. For instance, in one example, the physical environment to be“twinned” may be imaged using three-dimensional scanning techniques thatcapture the dimensions and layout of the physical environment, as wellas the dimensions and locations of any objects that are present in thephysical environment. The physical environment may be scanned using ahead mounted display or other imaging device. Virtual representations ofthe physical environment and/or objects may then be constructed from theimages. In another example, the dimensions and layout of the physicalenvironment (and optionally any objects present in the physicalenvironment) may be provided to the processing system (e.g., by a humanuser or by a database). In one example, the digital twin may besuperimposed onto an image of the physical environment, e.g., usingfiducials, recognized three-dimensional objects, or the like, in orderto verify that the dimensions and layout of the physical environmenthave been accurately rendered in the digital twin.

In one example, the digital twin may be constructed to include one ormore permanent or temporary modifications to the physical environment(without making the modifications to the physical environment in thereal world), e.g., so that the impacts of the modification can beevaluated. For instance, if the physical environment is a public placesuch as the entrance to an amusement park, the digital twin mightreplicate the dimensions and layout of the entrance, but might alsoinclude a proposed seasonal or holiday display that is positioned justinside the gates. If the physical environment is a home, the digitaltwin might replicate the dimensions and layout of a room within thehome, but might also change the location of a wall in the room. If thephysical environment is a manufacturing floor, the digital twin mightreplicate the dimensions and layout of the factory floor, but might alsorearrange the locations of one or more pieces of manufacturingequipment. In one example, the modification(s) may be defined by a user.

For instance, referring back to FIG. 3A, the example digital twin 300 ofthe amusement park entrance may modify the present layout of theentrance to include a holiday display 306. In the example of FIG. 3A,the holiday display 306 comprises a giant inflatable Halloweenjack-o-lantern. The holiday display 306 may comprise an item or itemsthat are meant to be deployed at the amusement park entrance temporarily(e.g., for a matter of weeks), rather than a permanent part of theentrance layout. However, in other examples, a permanent fixture (e.g.,a statue, a fountain, or the like) may be proposed for deployment at theentrance.

In step 206, the processing system may construct a social automaton fordeployment in the digital twin. As discussed above, the social automatonmay comprise a virtual representation of a person (e.g., an avatar)which may be deployed within the digital twin and may interact with thedigital twin and any objects within the digital twin, as well as withother social automatons that are present in the digital twin. The socialautomaton may represent a specific person or may represent a particulardemographic. In one example, the behaviors and/or attributes of thesocial automaton may be defined by a user and rendered by the processingsystem accordingly. In one example, step 206 may involve theconstruction of a plurality of social automatons for simultaneousdeployment in the digital twin. The plurality social automatons may allhave the same behaviors and attributes, or two or more social automatonsof the plurality of social automatons may have different behaviors andattributes (e.g., relative to each other).

For instance, as discussed above, a social automaton constructed inaccordance with step 206 may be programmed to interact with the physicalenvironment as a member of a particular demographic (e.g., an elderlyperson, a toddler, a person using a mobility aid such as a wheelchair orwalker, etc.). In this case, the social automaton may be programmed todemonstrate different behaviors and to exhibit different characteristicsbased on the demographic that the social automaton is intended torepresent. For instance, if the social automaton is programmed torepresent a toddler, then the social automaton may be short and unsteadyon its feet and may move quickly. If the social automaton is programmedto represent an elderly person, however, the social automaton may betaller and may move more cautiously.

In another example, the social automaton may be programmed to exhibitspecific behaviors in response to encountering certain objects or brandsof objects in the digital twin. For instance, a social automaton that isprogrammed to demonstrate the behaviors and exhibit the characteristicsof children may slow down its movement when passing an object thatrepresents a display device (e.g., a television, a computer monitor, amovie screen, or the like). By contrast, a social automaton that isprogrammed to demonstrate the behaviors and exhibit the characteristicsof an adult or an elderly person may be unaffected by the proximity tosuch objects.

FIG. 3B, for instance, illustrates an example overlay 302 comprising setof social automatons 308 ₁-308 _(n) (hereinafter individually referredto as a “social automaton 308” or collectively referred as “socialautomatons 308”) that may be constructed for insertion into the exampledigital twin 300 of FIG. 3A. In this case, the set of social automatons308 may be programmed to replicate the behaviors and characteristics ofvarious people who may be present in the amusement park, such as guests(e.g., families with children, groups of teenagers, etc.) and employees(e.g., photographers, ride operators, vendors, etc.). In one example,the number of social automatons 308 constructed for insertion into thedigital twin 300 (and the number of each demographic represented) maysimulate an estimated number of guests expected to be entering the parkduring a given window of time and/or on a given day (e.g., on a summerSaturday between 10:00 AM and 12:00 PM).

In one example, data modeling and analysis techniques may be used toconstruct the social automaton. For instance, recorded data (e.g.,images, video, etc.) of one or more individuals who belong to aparticular demographic may be analyzed and used as the basis formodeling a representative social automaton. In another example, apredictive service may construct the social automaton based on a name orother social links (if the social automaton is meant to represent aspecific individual). For instance, the general demographics, the socialmedia connections and/or activity, the profile data from one or moreservice providers, and/or the like for the specific individual may bemined for information that can be incorporated into the socialautomaton. For instance, if the specific individual's social mediaactivity indicates that the specific individual is a marathon runner, oris currently using crutches while recovering from an injury, then asocial automaton programmed to represent the individual may be tailoredaccordingly (e.g., the movements, speed, gait, or the like of the socialautomaton may be tuned to match the movements, speed, gait, or the likeof the specific individual). In another example, a social automaton thatis programmed to represent a specific demographic may be created from anaggregate movement (e.g., from historical opt-in gyroscopic and locationdata) or from more generalized statistics (e.g., a statistic indicatingthat the average adult typically walks at a pace of two to three milesper hour).

In one example, constructing the social automaton in accordance withstep 206 may involve retrieving the social automaton from a database ofstored social automatons. For instance, social automatons that wereconstructed for past simulations may be stored and used in futuresimulations to expedite processing. Optionally, modifications may bemade to a stored social automaton based on the specifications of theuser. For instance, a stored social automaton that represents a specificindividual may be modified to reflect a change to the specificindividual (e.g., an injury, an item carried by the specific individual,etc.).

In step 208, the processing system may insert the social automaton intothe digital twin to create an extended reality media in whichinteractions of the social automaton with the digital twin simulateinteraction of an individual (e.g., the individual or demographic thatthe social automaton is programmed to represent) with the physicalenvironment that is represented by the digital twin. For instance, inone example, the social automaton may be rendered as an overlay that issuperimposed on the digital twin. For instance, the overlay may bealigned with the digital twin (e.g., using fiducials or other alignmentmechanisms). As discussed above, step 208 may involve inserting aplurality of social automatons into the digital twin, where theplurality of social automatons may all have the same behaviors andcharacteristics, or at least two social automatons of the plurality ofsocial automatons may have different behaviors and characteristics(e.g., relative to each other).

For instance, referring again to the example of FIGS. 3A and 3B, theamusement park operators may wish to identify the optimal placement forthe holiday display 306 to minimize any impact on the flow of foottraffic at the park entrance. Thus, the set of social automatons 308 maybe programmed to replicate the behaviors and characteristics of variouspeople who may interact with or encounter the holiday display 306 insome way, such as guests (e.g., families with children, groups ofteenagers, etc.) and employees (e.g., photographers, ride operators,vendors, etc.). In one example, the number of social automatons 308constructed for insertion into the digital twin 300 (and the number ofeach demographic represented) may simulate an estimated number of parkpatrons expected to be entering the park at a given time and/or on agiven day.

FIG. 3C illustrates an extended reality media 304 in which the socialautomatons of FIG. 3B may be inserted into the digital twin 300 of FIG.3A. For instance, some guests may linger to take photos with theseasonal display 306 (e.g., social automatons 308 ₂-308 ₄), while otherguests may bypass the seasonal display 306 and continue through to othersections of the park (e.g., social automaton 308 ₁). A photographeremployed by the park may be stationed near the holiday display 306 inorder to take photos of guests (e.g., social automaton 308 _(n)). Stillother park employees may be positioned near the entrance to taketickets, check bags, and the like. The social automatons 308 may beprogrammed to simulate these behaviors when the social automatons 308are inserted into the digital twin 300. Inserting the social automatonsinto the digital twin and allowing the social automatons to interactwith the physical environment (including the holiday display 306) mayallow park operators to determine that, during peak crowd times, theholiday display 306 may inhibit the flow of guests into the park. Assuch, the park operators may consider moving the holiday display 306 toanother area of the park.

In one example, insertion of the social automaton into the digital twinmay be guided by one or more user-defined parameters including time tosimulate (e.g., time of day, day of week, season, or the like),situation to simulate (e.g., a special event occurring in the physicalenvironment such as a wedding versus a normal operating day), and/orobject to simulate (e.g., a new object or other change that is proposedto the physical environment and simulated in the digital twin). In someexamples, social automatons may be capable of interacting with eachother as well as with objects in the digital twin.

In optional step 210 (illustrated in phantom), the processing system maymake a modification to at least one of the digital twin and the virtualautomaton that is inserted into the digital twin. In one example, themodification may involve the addition, removal, or modification of anobject in the digital twin (e.g., addition or relocation of a piece ofequipment, etc.). In another example, the modification may involve themodification of the dimensions and/or layout of the digital twin (e.g.,removal or addition of a wall, etc.). In another example, themodification may involve the addition, removal, or modification of asocial automaton (e.g., adding another social automaton, changing thebehaviors and/or characteristics of an existing social automaton, etc.).Making the modification may involve repeating steps that were performedpreviously, such as imaging the physical environment, rendering thedigital twin and/or social automation, aligning an overlay including thesocial automaton with the digital twin, and/or other steps. Thus, steps204-208 may be repeated one or more times for evaluation of the samedigital twin.

In one example, the modification may be made in response to a requestfrom a user. For instance, the user may reconsider a particular changeto the physical environment based on the interaction of a socialautomaton within the digital twin of the physical environment. Inanother example, the modification may be made in response to a useraccepting a suggestion from the processing system. For instance, theprocessing system may suggest the deletion, addition, or modification ofa social automaton. The suggestion may be based on simulations thatother users have asked the processing system to perform in the past(e.g., x percent of users who have asked to simulate scenario A havealso asked to simulate scenario B).

In optional step 212 (illustrated in phantom), the processing system maysave the digital twin and the social automaton as a stored simulation.The stored simulation may help the processing system to render future,potentially similar digital twins and/or social automatons. The storedsimulation may also help the processing system to make recommendationsfor modifications to future digital twins and/or social automatons, asdiscussed above.

The method 200 may end in step 214.

The method 200 therefore allows a user to assess the dynamic impacts ofproposed changes to a physical environment. The proposed changes maycomprise physical changes to the physical environment itself (e.g.,changing the dimensions or layout of the physical environment) changesto objects in the physical environment (e.g., modifying, moving, adding,or removing objects), changes to a presence in the physical environment(e.g., more or fewer people present), or other changes. By insertingsocial automatons into a digital twin that represents the changedphysical environment, the user may be able to accurately simulate theeffects that the proposed changes will have on the interactions ofindividuals within the physical environment, without having to incur theeffort or expense to actually make the proposed changes in the physicalenvironment. This may allow the user to evaluate one or morealternatives to the proposed changes and to identify the best way tocarry out the proposed changes before making any actual changes to thephysical environment.

Thus, examples of the present disclosure leverage a fully virtualenvironment (e.g. both the physical environment and the individual(s)interacting with and in the physical environment may comprise virtualmodels) in order to test different scenarios for gaming, training,architectural and event planning, and other applications. The presentdisclosure therefor minimizes the costs and the time associated withtesting the scenarios in more conventional manners (e.g., implementingthe scenarios in the real world physical environment).

For instance, if the digital twin replicates a wedding venue (e.g., acatering hall, a hotel ballroom, a restaurant, or the like), then thesocial automatons may be programmed to replicate the behaviors andcharacteristics of different wedding guests. For instance, some adultguests may congregate near the bar area, while other younger guests maytend to gather near a photo booth. Guests of varying ages may formgroups on the dance floor. Based on insertion of these social automatonsinto the digital twin, a wedding planner might decide to set the bararea some distance away from the dance floor (e.g., so that therespective crowds do not overlap and form an even bigger crowd), torearrange the placement of guest tables, or to make other changes to thephysical environment of the wedding venue.

In another example, if the digital twin replicates a wing of a museum,then the social automatons may be programmed to replicate the behaviorsand characteristics of museum guests. For instance, some guests maylinger at a particular exhibit for a long time, while other guests maysimply walk past the exhibit. The digital twin may move the particularexhibit to a different location in the wing of the museum, and theinteractions of the social automatons with the relocated particularexhibit may be observed in order to determine whether moving theparticular exhibit to the different location improves the flow of foottraffic through the museum (e.g., minimizes the formation of crowds solarge that guests cannot get through to other sections of the museum).

Although the social automatons discussed above are described assimulating the behaviors and characteristics of people (or differentgroups of people), it will be appreciated that social automatons fornon-human beings may also be inserted into a digital twin withoutdeparting from the scope of the present disclosure. For instance, socialautomatons that are programmed to simulate the behaviors andcharacteristics dogs and cats may be inserted into the digital twin of ahome or business (e.g., a veterinary office, a pet store, a hospitalthat hosts therapy animals, etc.) in order to evaluate the safety ofintroducing the dogs or cats into the physical environment. In a similarexample, the social automatons may be programmed to representsemi-sentient IoT or robotic entities whose behavior is defined bynetwork and affinity descriptors (e.g., must have high bandwidth, mustuse minimal power, must always be adjacent to another entity, etc.). Inthis case, the network and affinity descriptors may comprise stand-insfor interaction simulations, instead of behaviors typical of livingentities.

Social automatons may also be programmed to simulate a specific butatypical behavior for a person, demographic, or other being. Forinstance, a plurality of social automatons may be programmed tospecifically simulate unruly behavior, for instance in order to evaluatethe effects of a potentially rowdy group of fans at a football game orconcert. Another social automaton could be programmed to simulate anunfriendly dog, in order to evaluate the potential effects of theunfriendly dog on the safety of a home.

Further examples of the present disclosure may insert social automatonsinto a digital twin in order to simulate and evaluate environmentalconditions that might be too dangerous or too difficult to test in thephysical environment, such as zero gravity conditions, fires, naturaldisasters, or the like.

Still further examples of the present disclosure may be used by atelecommunications service provider looking to determine where to placenetwork equipment. For instance, a plurality of digital twins comprisinga plurality of candidate sites for new 5G cells may be constructed.Social automatons may be inserted into the plurality of digital twins inorder to determine where the service provider's customers are mostlikely to require service.

In another example, social automatons may be inserted into gaming ortraining media. For instance, a social automaton could be programmed tobehave in a hostile manner or an excited manner, or could be programmedto have an affinity for a specific object in the gaming or trainingscenario (e . . . , trying to guard the object).

Although not expressly specified above, one or more steps of the method200 may include a storing, displaying and/or outputting step as requiredfor a particular application. In other words, any data, records, fields,and/or intermediate results discussed in the method can be stored,displayed and/or outputted to another device as required for aparticular application. Furthermore, operations, steps, or blocks inFIG. 2 that recite a determining operation or involve a decision do notnecessarily require that both branches of the determining operation bepracticed. In other words, one of the branches of the determiningoperation can be deemed as an optional step. However, the use of theterm “optional step” is intended to only reflect different variations ofa particular illustrative embodiment and is not intended to indicatethat steps not labelled as optional steps to be deemed to be essentialsteps. Furthermore, operations, steps or blocks of the above describedmethod(s) can be combined, separated, and/or performed in a differentorder from that described above, without departing from the examples ofthe present disclosure.

FIG. 4 depicts a high-level block diagram of a computing devicespecifically programmed to perform the functions described herein. Forexample, any one or more components or devices illustrated in FIG. 1 ordescribed in connection with the method 200 may be implemented as thesystem 400. For instance, a server (such as might be used to perform themethod 200) could be implemented as illustrated in FIG. 4.

As depicted in FIG. 4, the system 400 comprises a hardware processorelement 402, a memory 404, a module 405 for evaluating digital twinsusing social automatons, and various input/output (I/O) devices 406.

The hardware processor 402 may comprise, for example, a microprocessor,a central processing unit (CPU), or the like. The memory 404 maycomprise, for example, random access memory (RAM), read only memory(ROM), a disk drive, an optical drive, a magnetic drive, and/or aUniversal Serial Bus (USB) drive. The module 405 for evaluating digitaltwins using social automatons may include circuitry and/or logic forperforming special purpose functions relating to the operation of a homegateway or AR server. The input/output devices 406 may include, forexample, a camera, a video camera, storage devices (including but notlimited to, a tape drive, a floppy drive, a hard disk drive or a compactdisk drive), a receiver, a transmitter, a speaker, a display, a speechsynthesizer, an output port, and a user input device (such as akeyboard, a keypad, a mouse, and the like), or a sensor.

Although only one processor element is shown, it should be noted thatthe computer may employ a plurality of processor elements. Furthermore,although only one computer is shown in the Figure, if the method(s) asdiscussed above is implemented in a distributed or parallel manner for aparticular illustrative example, i.e., the steps of the above method(s)or the entire method(s) are implemented across multiple or parallelcomputers, then the computer of this Figure is intended to representeach of those multiple computers. Furthermore, one or more hardwareprocessors can be utilized in supporting a virtualized or sharedcomputing environment. The virtualized computing environment may supportone or more virtual machines representing computers, servers, or othercomputing devices. In such virtualized virtual machines, hardwarecomponents such as hardware processors and computer-readable storagedevices may be virtualized or logically represented.

It should be noted that the present disclosure can be implemented insoftware and/or in a combination of software and hardware, e.g., usingapplication specific integrated circuits (ASIC), a programmable logicarray (PLA), including a field-programmable gate array (FPGA), or astate machine deployed on a hardware device, a computer or any otherhardware equivalents, e.g., computer readable instructions pertaining tothe method(s) discussed above can be used to configure a hardwareprocessor to perform the steps, functions and/or operations of the abovedisclosed method(s). In one example, instructions and data for thepresent module or process 405 for evaluating digital twins using socialautomatons (e.g., a software program comprising computer-executableinstructions) can be loaded into memory 404 and executed by hardwareprocessor element 402 to implement the steps, functions or operations asdiscussed above in connection with the example method 200. Furthermore,when a hardware processor executes instructions to perform “operations,”this could include the hardware processor performing the operationsdirectly and/or facilitating, directing, or cooperating with anotherhardware device or component (e.g., a co-processor and the like) toperform the operations.

The processor executing the computer readable or software instructionsrelating to the above described method(s) can be perceived as aprogrammed processor or a specialized processor. As such, the presentmodule 405 for evaluating digital twins using social automatons(including associated data structures) of the present disclosure can bestored on a tangible or physical (broadly non-transitory)computer-readable storage device or medium, e.g., volatile memory,non-volatile memory, ROM memory, RAM memory, magnetic or optical drive,device or diskette and the like. More specifically, thecomputer-readable storage device may comprise any physical devices thatprovide the ability to store information such as data and/orinstructions to be accessed by a processor or a computing device such asa computer or an application server.

While various examples have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred example shouldnot be limited by any of the above-described example examples, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A method comprising: constructing, by aprocessing system comprising at least one processor, a digital twin of aphysical environment; constructing, by the processing system, a socialautomaton, wherein the social automaton comprises a virtualrepresentation of an individual that is programmed to exhibit behaviorsand characteristics of the individual; and inserting, by the processingsystem, the social automaton into the digital twin to create an extendedreality media in which interactions of the social automaton with thedigital twin simulate interactions of the individual with the physicalenvironment.
 2. The method of claim 1, wherein the digital twinreplicates a set of dimensions of the physical environment, a layout ofthe physical environment, and at least one object that is present in thephysical environment.
 3. The method of claim 1, wherein the digital twinincludes a modification to the physical environment, without making themodification to the physical environment.
 4. The method of claim 3,wherein the modification comprises a change to the set of dimensions. 5.The method of claim 3, wherein the modification comprises a change tothe layout.
 6. The method of claim 3, wherein the modification comprisesa change in a location of the at least one object.
 7. The method ofclaim 3, wherein the modification is a proposed permanent modification.8. The method of claim 3, wherein the modification is a proposedtemporary modification.
 9. The method of claim 1, wherein the individualis a specific individual.
 10. The method of claim 9, wherein the socialautomaton is constructed based on an analysis of social media contentassociated with the specific individual.
 11. The method of claim 1,wherein the individual is a representative of a demographic ofindividuals.
 12. The method of claim 11, wherein the social automaton isconstructed based on an analysis of recorded data depicting at least oneindividual of the demographic.
 13. The method of claim 1, wherein thesocial automaton is one of a plurality of social automatons that issimultaneously inserted into the digital twin.
 14. The method of claim13, wherein interactions of the plurality of social automatons with eachother in the digital twin are simulated by the extended reality media.15. The method of claim 13, wherein at least two social automatons ofthe plurality of social automatons are programmed to exhibit differentbehaviors and characteristics.
 16. The method of claim 1, wherein thesocial automaton is rendered by the processing system as an overlay, andthe inserting comprises superimposing the overlay over the digital twin.17. The method of claim 1, further comprising: modifying, by theprocessing system at least one of the digital twin and the socialautomaton, subsequent to the inserting.
 18. The method of claim 17,wherein the modifying is performed in response to a user request.
 19. Anon-transitory computer-readable medium storing instructions which, whenexecuted by a processing system including at least one processor, causethe processing system to perform operations, the operations comprising:constructing a digital twin of a physical environment; constructing asocial automaton, wherein the social automaton comprises a virtualrepresentation of an individual that is programmed to exhibit behaviorsand characteristics of the individual; and inserting the socialautomaton into the digital twin to create an extended reality media inwhich interactions of the social automaton with the digital twinsimulate interactions of the individual with the physical environment.20. A device comprising: a processor; and a computer-readable mediumstoring instructions which, when executed by the processor, cause theprocessor to perform operations, the operations comprising: constructinga digital twin of a physical environment; constructing a socialautomaton, wherein the social automaton comprises a virtualrepresentation of an individual that is programmed to exhibit behaviorsand characteristics of the individual; and inserting the socialautomaton into the digital twin to create an extended reality media inwhich interactions of the social automaton with the digital twinsimulate interactions of the individual with the physical environment.